CollectiveX: kv-transfer suite — NIXL + MoRI-IO KV-cache handoff benchmark (stacked on #2489) - #2510

Slurm propagates the submitter's soft RLIMIT_MEMLOCK into job steps, and a
3.8 GiB soft limit fails a 12.7 GB pool MR deep inside the library (MoRI
errno 12, UCX EIO on ucp_mem_map) with no hint at the cause. Raise soft to
hard when hard allows -- covering every launcher path without touching
their salloc flags -- and fail with the actual numbers when it does not.

Validated on the metal over the Pollara rails: 24-25 GB/s paged at library
defaults (qp=1, no chunking), pattern-verified on every row. NIXL stays off
this SKU for now -- both the image-bundled and a source-built UCX stack fall
back to TCP (~0.8 GB/s) for ROCm memory on the ionic provider; the GDR path
there is still being diagnosed and the row would misrepresent the library.

UCX auto-selection is a wrong-fabric trap (b200-nscale's quad-port aux
card, b300's storage IB). run_kv maps COLLX_RDMA_DEVICES /
COLLX_IB_GID_INDEX into UCX_NET_DEVICES / UCX_IB_GID_INDEX before any
backend instantiates UCX; explicit UCX_* values still win.

The probe finally scheduled once the request matched what a KV leg needs
(gpu:1 + bounded mem on a mixed partition): 36/36 rows pattern-verified
over the IB VFs -- bulk 43.4 GB/s, paged-64 32-36, paged-16 9.2-16.2 (the
per-descriptor floor rides a little heavier on SR-IOV).

Mooncake TransferEngine, probe-validated on b200-nscale/h200/gb200 (36/36
verified rows each; its batch path has the lowest per-descriptor floor and
beats NIXL at 16-token MLA pages on B200) and measured infeasible on AMD
(the wheel links libcuda.so.1 at import). The adapter dlopens
libcudart.so.12 from nvidia-cuda-runtime-cu12, so cu13 images need no
LD_LIBRARY_PATH seam.

Fabrics become real: pools move behind kv_pool (torch for rdma, cuMem
FABRIC via ctypes for mnnvl -- UCX's cross-node cuda_ipc only engages on
fabric-mappable memory; on cudaMalloc the flag is inert and silently rides
the IB rails), adapters register raw pointers, and gb200 gains the mnnvl
row. Measured on the rack: bulk 636-707 GB/s (7.9x the IB rails) but ~3.9us
per descriptor copy, so paged rows land BELOW the IB lane -- the inversion
the two fabric rows exist to publish.

run_kv smoke-ran end to end on b200 (nixl + mooncake, rdma) and gb200
(nixl, mnnvl): status=success artifacts on all three. The smoke caught a
bulk-row verdict crash (no verifying side -> StopIteration on both ranks),
fixed as exchange_verdict with the no-verifier case pinned by test. Also
trims prose to its factual core and drops the dead --link-gbps parameter.

…date gb-nv rdma legs

Review findings on #2510, both real. The launchers collx_die on COLLX_BENCH
values outside their EP enum, so every kv shard died at the identity stage;
nixl/mooncake/mori-io are now accepted where the registry schedules them,
pinned by a test that greps each SKU's launcher for its kv backends.

launch_gb-nv.sh also exported COLLX_TRANSPORT=mnnvl unconditionally, which
made a gb200 kv rdma leg the only scale-out fabric that skipped
collx_apply_network_profile, the rank wrapper's network branch, and
validate_container_network. kv rdma shards now carry mnnvl-rdma (the
workflow exports the shard mode) and the launcher proves the pinned socket
interface and HCAs on the allocation before running, like every other
scale-out launcher; mnnvl shards keep skipping, as elsewhere.

Per the model config: the MLA cache is expressed as MQA (1 kv head x
head_dim 512) plus rope 64, and the DSA indexer keeps its own k cache
(index_head_dim 128) that vLLM's connector merges into the transfer
regions, so the per-token-per-layer transfer unit becomes 704 elements
across the same 61 layers. Workload name and case ids stay kv-mla; the
probe evidence in the PR was measured at the prior 576-element shape and
the first CI dispatch re-baselines at this one.

…cross batch sizes

kv-dsv4 replaces kv-mla: V4-Pro has no MLA; the preset transcribes vLLM's
cache specs region by region (30 CSA layers at 4 tokens per 576B fp8_ds_mla
entry plus their 132B lightning-indexer entries, 31 HCA layers at 128 tokens
per entry, and the 128-token sliding-window cache on all 61 layers), pinned
fp8 because the dtype mix is architectural. Transfers gain a batch dimension:
each request in a burst is its own prepped transfer over a disjoint slice of
one block-table permutation; bursts post all requests then await all, timed
as one completion (split post/wait backend contract). Grid points shed
batches that cannot fit the per-rank pool budget instead of dropping out.
Mooncake posts its sync calls from a worker pool, the SGLang connector shape.
Verification is per-byte so unaligned regions (132B indexer) check exactly.

The kv precision gate moved sweep_matrix onto kv_workload, which imports
numpy at module top; the matrix/extract steps run on bare runners where that
import is not a given. The sweep config now maps each workload to its
precisions directly (a test pins the map to the workload model's PRESETS,
and plan_config still fail-closes on a mismatch at runtime). The CollectiveX
test job installs numpy: the cpu torch wheel does not pull it in, so every
kv test module had been failing at import since the suite landed.

A kv-transfer validation run had no way in: the default dispatch resolves
every suite (the full EP matrix rides along), and only_sku still drags that
SKU's EP shards. The input passes straight to sweep_matrix --suites and
joins the concurrency key so suite-scoped runs do not queue behind full
sweeps.

The noble-based sweep images mark the container python externally managed
(PEP 668), so the bare mooncake install refused on b200 and gb200 in the
first kv CI run (nixl dodged it only because the image bundles nixl). Same
try-then-retry shape the uccl prep already uses; older pips never refuse,
so they never reach the flag.

… pairs

On a same-rack GB200 pair the engine's NVLink-IPC transport claims the
cross-node segments (one NVLink domain) and fails the address import
(nvlink_transport 'Requested address not found', first kv CI run). The row
declares the rdma lane, so the adapter sets MC_USE_NVLINK_IPC=0; the ROCm
twin knob (MC_USE_HIP_IPC) misclaims the same way on mi355x. Transfer
failures now carry the library rc.

launch_gb-nv.sh exported MC_FORCE_MNNVL=1 for every bench; mooncake is its
only reader and it responds by installing only the cross-node NVLink
transport, which cannot open another host's segments in the pinned wheel
(cudaIpcOpenMemHandle: invalid resource handle; found 6 HCAs but skipped
rdma, kv CI runs 1-3 on gb200). The mooncake kv row declares the rdma lane,
so the mooncake bench opts out; EP benches keep the export untouched.

Drops the kv-gqa preset and its cases: the suite measures the deployment
shape of interest, and the dense-counterpoint numbers told their story (the
descriptor-floor gap is visible against each lane's own bulk ceiling). One
workload x one precision per shard; the pool-budget shedding test pins the
mechanism against a synthetic budget since dsv4 alone never nears 64 GiB.

run_kv copied its --version argv string straight into the document while
ep_harness types the same flag as int; consumers comparing against the
numeric matrix version had to coerce. Mirrors the EP entrypoint.

The first production charts show a bandwidth bump at batch 4 on some lanes
where scaling should be monotonic; batches 2 and 8 bracket it to separate
a real concurrency sweet spot from sampling noise, and a fourth trial
tightens the p50. Worst lane (gb200 mnnvl descriptor floor) computes to
about 31 minutes per case, well inside the 5400 s case guard.

…ptor budget

The ladder becomes 8k / 32k / 128k / 512k. A 512k page-16 request alone is
~2.1M descriptors and burst posting time is linear in batch x descriptors
on the per-descriptor floor, so points now shed batches whose burst exceeds
DESC_BUDGET (the smallest batch always survives, keeping a single request
measurable at every point) before the pool-budget fit; re-planning the pool
for the surviving batch also drops the 512k pool from ~57 GB to ~8 GB. The
budget is sized to keep every 32k cell of the previous grid; the fourth
sampling trial reverts to three now that the dense grid has pinned the
mooncake batch-4 peak as stable signal. Slowest lane (gb200 mnnvl floor)
computes to ~62 minutes per case against the 5400 s guard.

One octave past the mooncake collapse (does batch 32 keep falling?), past
nixl's flat line, and toward mori-io's saturation. The descriptor budget
sheds the new rung wherever a burst cannot afford it (32k page-16 stays
capped at 16, the 128k/512k caps are unchanged), so only 8k and 32k page-64
points grow; the slowest lane computes to ~73 minutes against the 5400 s
guard, and the largest pool stays ~7.4 GB.

The descriptor budget places the new rung where a burst affords it (8k both
pages; 32k page-64 keeps 32, larger ISLs unchanged). The gb200 mnnvl
descriptor floor makes the dense kv grid ~90 minutes of legitimate work, so
kv benches on gb-nv raise the per-case hang guard to 7200 s; every other
lane computes to 45 minutes or less under the default guard.

A kv case is ~45 minutes of work (x86) or ~90 (gb mnnvl floor) but the
allocation asked for the fleet-wide 300; on a contended pool a 5-hour
exclusive 2-node ask cannot backfill into gaps between long benchmark jobs
and pends for hours (both h200 legs burned two full GHA budgets waiting
behind the pool's post-wedge backlog). kv benches now ask 150 (single-slurm)
or 180 (gb-nv) minutes, which Slurm can slot into windows the big jobs
leave open.

Every other CI launcher submits with --job-name="$RUNNER_NAME" so
operators can squeue/scancel a runner's work by name; CollectiveX
allocations carried Slurm's default name. collx_salloc_jobid now passes
the runner name when the Actions runner provides it. Hand launches
without RUNNER_NAME keep the default, and the launcher's own cleanup
still tracks the allocation by job id.

The frontier chart draws its line through the batch ladder at the largest
measured ISL, and descriptor-budget shedding left that ladder 2-3 points
(524288/p16 kept only [1,2]). Raise the always-survive floor from the two
smallest batches to LADDER_FLOOR=5 so every (isl, page) point stays
chartable: the largest-ISL cells now carry [1,2,3,4,6] and 131072/p16
grows from 4 to 5 rungs, while cells the budget already served keep their
ladders unchanged.

The floor prices at ~1.63x descriptor work grid-wide (worst burst is
524288/p16 batch 6, ~5.6x DESC_BUDGET, bounded); recompute the three kv
launcher budgets from their measured anchors: gb-nv 240/13200 -> 420/22800
(mnnvl descriptor floor, ~350 min projected), single-slurm and mi-amds
180/9000 -> 210/11400 (~170 and ~145 min projected).

On the power-of-two ladder the descriptor-floor lanes post tens of
millions of descriptors per burst, so one grid point's timed stretch can
exceed gloo's 30 minute default recv timeout; the target rank then dies
at the next gather while the initiator is still doing honest work (gb200
nixl-mnnvl, run 33137809635, Timed out waiting 1800000ms for recv).
Initialize the process group with a timeout taken from COLLX_RUN_TIMEOUT
so the per-case hang guard, not the control plane, decides when a run
has failed.

The power-of-two batch ladder plans a 53 GiB pool at 512k ISL (pool is
sized for the largest surviving batch, now 16 instead of 6), and
mooncake's transfer engine cannot register that on the ionic NICs:
ibv_reg_mr fails with ENOMEM, deterministically, on two independent
allocations (runs 33137809635 and 33150394862). mori-io registers the
same pool fine, so the wall is the engine/NIC pairing, not the ladder.
Make POOL_BUDGET launcher-overridable (COLLX_KV_POOL_BUDGET, bytes) and
cap the mi355x mooncake leg at the 20 GiB the mixed ladder proved green;
only the two 512k points shed to a three-rung ladder, every other point
keeps its full ladder.

…ation

The kv gb200 mnnvl leg holds a 460 minute Slurm allocation with a 420
minute per-case guard inside it, but the GitHub job ceiling was still
350 minutes, so run 33150394862 was cancelled at 350 minutes while
doing honest work (the gloo control-plane fix held; the leg simply
needs ~380). Order the ladder correctly: job ceiling 480 > allocation
460 > per-case guard 420 > ~380 projected. EP shards finish far
earlier and are unaffected.

The gb300 trays carry four ConnectX-8 XDR800 InfiniBand rails (mlx5_0-3,
all ACTIVE; cross-node RC pingpong ~9us LID-routed), so the SKU gets the
same three kv lanes as gb200: nixl over rdma and mnnvl, mooncake over
rdma. EP shards stay MNNVL-only and never touch the NICs; the new network
block only feeds the kv rdma legs' fail-closed profile and validation.
The launcher already accepts gb300 for the kv benches with the 460-minute
allocation and 420-minute guard, and batch_1_qos carries no MaxWall.

The gb300 nixl-mnnvl leg paces ~1.8x gb200 at isl >= 131072, reproduced
across two node pairs (run 33244478580 on c007/c008 and the 2026-08-29
hand retest on c004/c006 agree at 524288, ~13.7s vs gb200's 7.5s per
transfer at batch 1), so the full pow2 grid projects ~600 minutes and
both prior CI legs died at the 420 minute guard while doing honest work.
Give gb300 kv legs a 690 minute allocation with a 660 minute guard and
lift the workflow ceiling to 720 so GitHub cannot cancel a healthy leg
first. gb200 keeps 460/420.

b300 gets the same two kv lanes as b200-nscale: nixl and mooncake over
rdma. The existing network block (bond0, gid 3, rail NICs) already feeds
the EP scale-out shards, and the single-slurm launcher already accepts
the nixl and mooncake benches for b300, so capability is the only gap.
No mnnvl lane: b300 is an 8-GPU NVLink node, not a rack-scale domain.

The old workload model measured a synthetic shape vLLM never posts: 576 B
per compressed entry with per-entry alignment, and every (layer, page)
exploded into its own descriptor (up to ~2.1M descriptors per 512k request).
vLLM's packed DSV4 NIXL path registers ONE contiguous descriptor per packed
physical block per cache group, with 584 B token-states padded to a 576 B
multiple at page granularity (validated against vLLM 32ad1400d7). The old
shape inflates descriptor counts by two orders of magnitude, which can
invert backend and fabric conclusions on descriptor-bound lanes.

- kv_workload: regions are vLLM cache groups (c4a, c4a-idx, c128a, swa);
  block-major [block][layer] layout, packed_bytes = layers x page_bytes per
  descriptor; the sliding window's block is fixed at 64 tokens (shares the
  CSA tensor, page equals the CSA page byte for byte); block sizes that
  split an HCA state fail closed; grid runs the production block 256.
- kv_backend: time_bursts also records each request's host-observed
  completion offset from burst start (request_ms), so per-request latency
  is measured, never derived from burst p95 / batch.
- run_kv: rows carry request_ms and gbps_p50_incl_prep (the cold-path rate
  with prep paid once, for unique block tables and handle churn); sampling
  raised to 48 burst samples per row (2:16:3) now the packed geometry
  affords it, so p95 is no longer the max of 16.
- summarize: contig column renamed from the wire-ceiling framing to the
  contiguous baseline (host-observed goodput, not proven wire utilization).
- methodology: geometry section rewritten to the packed model; stale GB200
  per-lane bandwidth figures from the retired geometry removed (pending
  re-measurement), keeping only the descriptor-cost statement that matches
  live data; 16-token-page lane facts retired with the block sizes.
- tests re-pinned to the packed arithmetic by hand (block 256: c4a page
  37,440; packed 1,123,200; 6,146 descs per 512k request).

…sitive UCX_TLS

b300 ships UCX_TLS=rc cluster-wide in /etc/environment and srun --export=ALL forwards it into the container. An RC-only positive list makes ucp close the cuda mds, UCX then classifies VRAM as host memory, and NIXL registerMem fails with NIXL_ERR_BACKEND on every case. Extend such a list with cuda_copy,cuda_ipc instead of overriding it: the wire restriction stays the operator's choice, negation lists and 'all' already cover cuda, and explicit lists that name a cuda transport pass through untouched.

…extending it

Extending the positive list with cuda_copy,cuda_ipc regains registration
but the initiator then segfaults in ucp_worker_add_rkey_config resolving
the cuda rkey on the first ucp_get_nbx. Removing the inherited list lets
UCX auto-select transports; the wire remains pinned via UCX_NET_DEVICES.

b300's NICs refuse cuda memory registrations somewhere between 7083 and
8847 MiB. UCX surfaces no error at registration time; the initiator later
segfaults in ucp_worker_add_rkey_config resolving the region's rkey on the
first GET, so a full dsv4 grid (14 GiB pool) dies on its first point while
small pools pass.

Fix: the NIXL adapter registers the pool in pieces of at most 4 GiB. Cut
points must fall between transfer descriptors for every planned config at
once, so run_kv._harmonize first rewrites all configs onto one shared pool
layout (each region sized to the largest pool_blocks any config plans),
making region bases config-invariant; each region is then cut on its own
packed-block grid. Configs planning a different page size get their own
slab. The other adapters accept and ignore the layout.

…wall

Piece-wise registration lifted the ~8 GiB single-MR wall (an 8847 MiB pool
now runs green where whole registration segfaulted), but the same pods also
stop honoring cuda registrations past a per-rank total somewhere between
9552 MiB (green) and 14843 MiB (red), again with no registration-time error
and the same first-transfer segfault, whether the pool is one MR or six.
BAR1 is 512 GiB, so this is DMA mapping capacity, not aperture, and no
registration shape fixes it. Cap the b300 pool budget at 8 GiB the way the
mi355x mooncake launcher already caps its ionic ENOMEM wall; run_kv sheds
the largest batches to fit and every other grid point keeps its ladder.

…anism

Every kv case document now records the hostnames of both ranks
(topology.hosts) and the engine NIC filter actually applied
(implementation.nic_filter), and the mooncake adapter resolves its
library version across the dist names image-provided builds use, so a
row carries the evidence of which nodes, which rail, and which engine
build produced it. The b300 nixl pull-side spread between otherwise
identical runs is currently unattributable because none of that was
recorded.

Also corrects the mechanism stated in the previous commit: the ~1 s
cross-rail stall quantum is the worker pool's inactive-endpoint hold
(inactiveTime() > 1.0 before re-establishment), not the handshake
socket's receive timeout, which is 60 s in the image's engine and never
fires. The methodology now documents the b300 mooncake row as a
one-rail measurement.

b300 nixl pull degrades reproducibly across node pairs (three runs, two
distinct pairs: pull medians 38-45 GB/s with p95/p50 tails past 10x)
while push holds line rate, and the code between the flat-81 run and the
degraded ones did not touch nixl. The asymmetry points at the RDMA READ
path under UCX's own multi-rail selection on the rail-isolated pods, so
the registry's kv_device now also applies to UCX-backed cases: run_kv
hands it to UCX_NET_DEVICES verbatim (explicit UCX env still wins), the
nixl row records it as implementation.nic_filter, and b300 nixl pins
mlx5_0. If the pinned run holds a tight one-rail plateau, multi-rail
selection is the variance source; if the tails survive, the rail itself
is degraded and this becomes an SRE handoff.

…VICES

The pinned b300 run came back with two-rail line-rate push and pull
points byte-identical to the unpinned run: b300 forwards a blanket
16-device UCX_NET_DEVICES from /etc/environment through srun
--export=ALL, and the explicit-env-wins contract let it swallow both
the operator inventory and the new pin, so UCX has been self-selecting
on b300 all along. A registry pin is per-case operator intent, so it
now outranks the inherited value, the same way the inherited
UCX_TLS=rc is dropped; the inventory path without a pin still defers
to pre-set UCX env.

The renderer reads EP-suite row fields (routing, tokens_per_rank); a kv
shard's documents crashed it with KeyError 'routing' in every kv run's
summary step (non-fatal, the table just went missing).

The pinned run holds 47.7-49.0 GB/s both directions with p95/p50 at 1.01
across all 70 rows, against unpinned READs wandering 18-83 GB/s across
runs and node pairs. b300 kv rows are one-rail measurements for both
backends and directly comparable.